Weight saliency regularization in augmented networks

This paper introduces the concept of “optimally distributed computation” in feed-forward neural networks via regularisation of weight saliency. By constraining the relative importance of the parameters, computation can be distributed thinly and evenly throughout the network. We propose that this will have beneficial effects on fault tolerance performance and generalisation ability in augmented network architectures. These theoretical predictions are verified by simulation experiments on two problems — one artificial and the other a “real world” task. In summary, this paper presents regularisation terms for distributing neural computation optimally.